Concerning the demands of large capacity and high spectrum utilization in future mobile communication system, a method for optimizing the capacity of secondary user system in Multiple-Input Multiple-Output (MIMO) cognitive networks based on Non-Orthogonal Multiple Access (NOMA) was proposed. Firstly, the transmitted signals were pre-coded, and then the cognitive users were clustered according to channel gains. Secondly, the power allocation was performed for users after clustering. Finally, the Non-deterministic Polynomial-hard (NP-hard) multi-cluster objective function was transformed into solving the capacity of each sub-cluster. Meanwhile, taking into account Quality of Service (QoS) of cognitive users and requirement of Successive Interference Cancellation (SIC), the optimal power allocation coefficient, which is a constant between 0 and 1, was solved by using Lagrange function and Karush-Kuhn-Tucker (KKT) condition. The simulation results show that, the proposed method outperforms the average power allocation method. And when the channel quality is poor, compared with the MIMO cognitive network based on Orthogonal Multiple Access (OMA), the proposed method has improved the capacity of secondary user system significantly.